New <video> Policies for iOS

Since before your sun burned hot in space and before your race was born, Safari on iOS has required a user gesture to play media in a <video> or <audio> element. When Safari first supported <video> in iPhoneOS 3, media data loaded only when the user interacted with the page. But with the goal of returning more control over media playback to web developers, we relaxed this restriction in iOS 8: Safari began honoring the preload="metadata" attribute, allowing <video> and <audio> elements to load enough media data to determine that media’s size, duration, and available tracks. For Safari in iOS 10, we are further relaxing this user gesture requirement for silent <video> elements.

Motivation

It turns out that people these days really like GIFs. But the GIF format turns out to be a very expensive way to encode animated images when compared to a modern video codec like H.264. We’ve found that GIFs can be up to twelve times as expensive in bandwidth and twice as expensive in energy use. It’s so expensive that many of the largest GIF providers have been moving away from GIFs and toward the <video> element. Since most of these GIFs started out their lives as video clips, were converted into animated GIFs, and were again converted back to video clips, you might say that the circle is complete.

But while this move does spare websites’ bandwidth costs as well as saving users’ batteries, it comes at a usability cost. On iOS 9, <video>s will only begin playing as a result of a user gesture. So pages which replace an <img> with a <video> will require a user gesture before displaying their animated content, and, on iPhone, the <video> will enter fullscreen when starting playback.

Similarly, web developers are doing some seriously amazing stuff by integrating <video> elements into the presentation of their pages. And these pages either don’t work at all on iOS due to the user gesture requirement, or the <video> elements completely obscure the page’s presentation by playing an animated background image in fullscreen.

WebKit’s New policies for video

Starting in iOS 10, WebKit relaxes its inline and autoplay policies to make these presentations possible, but still keeps in mind sites’ bandwidth and users’ batteries.

By default, WebKit will have the following policies:

• <video autoplay> elements will now honor the autoplay attribute, for elements which meet the following conditions:
  • <video> elements will be allowed to autoplay without a user gesture if their source media contains no audio tracks.
  • <video muted> elements will also be allowed to autoplay without a user gesture.
  • If a <video> element gains an audio track or becomes un-muted without a user gesture, playback will pause.
  • <video autoplay> elements will only begin playing when visible on-screen such as when they are scrolled into the viewport, made visible through CSS, and inserted into the DOM.
  • <video autoplay> elements will pause if they become non-visible, such as by being scrolled out of the viewport.
• <video> elements will now honor the play() method, for elements which meet the following conditions:
  • <video> elements will be allowed to play() without a user gesture if their source media contains no audio tracks, or if their muted property is set to true.
  • If a <video> element gains an audio track or becomes un-muted without a user gesture, playback will pause.
  • <video> elements will be allowed to play() when not visible on-screen or when out of the viewport.
  • video.play() will return a Promise, which will be rejected if any of these conditions are not met.
• On iPhone, <video playsinline> elements will now be allowed to play inline, and will not automatically enter fullscreen mode when playback begins.
  <video> elements without playsinline attributes will continue to require fullscreen mode for playback on iPhone.
  When exiting fullscreen with a pinch gesture, <video> elements without playsinline will continue to play inline.

For clients of the WebKit framework on iOS, these policies can still be controlled through API, and clients who are using existing API to control these policies will see no change. For more fine-grained control over autoplay policies, see the new WKWebViewConfiguration property mediaTypesRequiringUserActionForPlayback. Safari on iOS 10 will use WebKit’s default policies.

Examples

So how would the humble web developer take advantage of these new policies? Suppose one had a blog post or article with many GIFs which one would prefer to serve as <video> elements instead. Here’s an example of a simple GIF replacement:

<video autoplay loop muted playsinline>
  <source src="image.mp4">
  <source src="image.webm" onerror="fallback(parentNode)">
  <img src="image.gif">
</video>

function fallback(video)
{
  var img = video.querySelector('img');
  if (img)
    video.parentNode.replaceChild(img, video);
}

On iOS 10, this provides the same user experience as using a GIF directly with graceful fallback to that GIF if none of the <video>‘s sources are supported. In fact, this code was used to show you that awesome GIF.

If your page design requires different behavior if inline playback is allowed vs. when fullscreen playback is required, use the -webkit-video-playable-inline media query to differentiate the two:

<div id="either-gif-or-video">
  <video src="image.mp4" autoplay loop muted playsinline></video>
  <img src="image.gif">
</div>

#either-gif-or-video video { display: none; }
@media (-webkit-video-playable-inline) {
    #either-gif-or-video img { display: none; }
    #either-gif-or-video video { display: initial; }
}

These new policies mean that more advanced uses of the <video> element are now possible, such as painting a playing <video> to a <canvas> without taking that <video> into fullscreen mode.

  var video;
  var canvas;

  function startPlayback()
  {
    if (!video) {
      video = document.createElement('video');
      video.src = 'image.mp4';
      video.loop = true;
      video.addEventListener('playing', paintVideo);
    }
    video.play();
  }

  function paintVideo()
  {
    if (!canvas) {
      canvas = document.createElement('canvas');
      canvas.width = video.videoWidth;
      canvas.height = video.videoHeight;
      document.body.appendChild(canvas);
    }
    canvas.getContext('2d').drawImage(video, 0, 0, canvas.width, canvas.height);
    if (!video.paused)
      requestAnimationFrame(paintVideo);
  }

<button onclick="startPlayback()">Start Playback</button> 

The same technique can be used to render into a WebGL context. Note in this example that a user gesture–the click event–is required, as the <video> element is not in the DOM, and thus is not visible. The same would be true for a <video style="display:none"> or <video style="visibility:hidden">.

We believe that these new policies really make video a much more useful tool for designing modern, compelling websites without taxing users bandwidth or batteries. For more information, contact me at @jernoble, Jonathan Davis, Apple’s Web Technologies Evangelist, at @jonathandavis, or the WebKit team at @WebKit.